Accept valid XML documents the token scan wrongly rejected - #175
Merged
Conversation
The root-element scan held the SAX-parity line too tightly in three places, rejecting documents a namespace-aware SAX parser accepts — safe-direction misses, but real gaps in the advertised parity: * Character references admit unlimited leading zeros (the CharRef grammar has no digit cap), so 	 is legal. Digit runs are now unbounded in the scanner and bounded lexically — significant digits only — before conversion, never by converting an unbounded digit run. * PI targets and entity names use the full XML Name grammar, where the colon namespace processing claims in QNames and prefixes is an ordinary name character: <?p:x?> and a DTD-covered &p:x; are valid. NCName still governs QName parts and xmlns prefixes. * Legacy multibyte encodings may use ASCII bytes as trail bytes (Shift_JIS ソ is 83 5C, ending in an ASCII backslash), derailing the raw byte scanner inside names, values and DOCTYPE subsets. Such documents are now replace-transcoded to UTF-8 up front; a U+FFFD in the consumed prefix marks input the parser would reject, while replacements past the root — trailing garbage, a character split by the scan limit — stay harmless, preserving the pinned boundary. With every scanned document UTF-8, valid_name? no longer threads the document encoding and valid_text? takes a text mode instead.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #173. The root-element scan held the SAX-parity line too tightly in three places, rejecting documents a namespace-aware SAX parser accepts — safe-direction misses, but real gaps in the advertised parity, confirmed against OpenJDK 17's namespace-aware SAX parser:
Leading zeros in character references. The CharRef grammar has no digit cap, so
	is legal. Digit runs are now unbounded in the scanner and bounded lexically — at most seven significant decimal digits or six hex, covering every code point through U+10FFFF — before conversion, so an unbounded digit run is never converted to an integer. All zeros denote #x0, still rejected.PI targets and entity names use the full XML Name grammar. Namespace processing claims the colon in QNames and prefixes only; in a PI target or entity name it is an ordinary name character, so
<?p:x?>and a DTD-covered&p:x;are valid. A newXML_NAMEgrammar (the NCName classes plus colon) is routed throughskip_processing_instructionandvalid_references?; QName parts andxmlns:prefixes stay NCName, and the reserved-xml-target check is untouched.Legacy multibyte encodings are transcoded before scanning. Shift_JIS ソ is
83 5C— its trail byte is an ASCII backslash — so the raw byte scanner stopped mid-character inside names, attribute values, and DOCTYPE subsets. Documents declaring any encoding other than UTF-8 are now replace-transcoded to UTF-8 up front; a U+FFFD in the consumed prefix (through the end of the root start-tag) marks input the parser would reject, while replacements past the root — trailing garbage, a character split byMAX_SCAN— are never examined, preserving the pinned "invalid bytes after the root are harmless" boundary. One deviation from the narrow fix: declared US-ASCII also takes the transcode path rather than staying raw, which is behavior-identical (ASCII transcodes 1:1; high bytes become U+FFFD and reject exactly as the oldvalid_encoding?check did; US-ASCII cannot encode a genuine U+FFFD) and leaves the raw path purely UTF-8. The rare charset that can encode U+FFFD itself, like GB18030, only forfeits a refinement — noted in a comment.With every scanned document UTF-8,
valid_name?no longer threads the document encoding andvalid_text?takes a text mode instead.Probes cover each gap in both directions — the valid documents now refine, and the invalid neighbors (all-zero scalars, oversized references, undeclared
&p:x;, bytes invalid in the declared encoding, US-ASCII high bytes) still hold the generic type — plus the boundary pins: invalid bytes stay fatal before the root and harmless after it, including a multibyte character split exactly at the scan limit. Full suite: 664 runs, 1684 assertions, 0 failures.